This invention generally relates to information processing systems, and more specifically, to a neural network type of information processing system that couples a set of input signals into one or more output signals according to a particular algorithm.
Neural networks are designed to simulate, in a very simplified way, the operation of the human brain; and in a neural network, a multitude of input terminals are coupled to a multitude of output terminals, referred to as neurons, by a multitude of data transfer members. For instance, a particular input terminal may be coupled to one, several, or all of the neurons of the network by the data transfer members; and, conversely, a particular neuron may be coupled to one, several, or all of the input terminals by the data transfer members. Typically, each output terminal generates an output signal that is some function of the sum of weighted values of the input signals coupled to the output terminal, and the weight that is given to a particular input signal by a given output terminal is determined by the data transfer member coupling that input signal to that output terminal.
In many neural networks, the weights that are given to the input signals may change over time. For example, for various applications, certain input signals should produce known output signals; and when such input signals are applied to a neural network, the actual output of the network can be compared to what it should be. If the actual output differs from what it should be, the weights that are applied to the input signals may be adjusted to bring the actual output of the network closer to, or equal to, what it should be.
As a general rule, when the number of neurons in a neural network is increased, the network can handle more complex tasks and can better simulate human thinking. Thus, considerable attention has been given to designing neural networks having a large number of neurons; and for example, neural networks have been designed with several thousand neurons. As will be appreciated, for reasons of economy and reliability, it is highly desirable to manufacture neural networks as integrated circuits.
Various practical factors, such as the large number of physical connections between the parts of a neural network, have heretofore limited the number of neurons that can be located in a neural network of a given size. To elaborate, commonly, each input signal of a neural network may be coupled to most or even all of the output terminals of the network by a respective one data transfer member. Thus, for example, a network having 200 input signals and 200 output terminals might have as many as 40,000 data transfer members, with each data transfer member being physically connected to one input terminal and one output terminal.
Moreover, in neural networks in which adjustable weights can be applied to the input signals, usually each, or most, of the data transfer members is physically connected to a respective control means that controls the weight the data transfer member applies to the input signal conducted to it. Each of these control means includes a multitude of components, such as a variable signal generator that applies a variable control signal to the associated data transfer member, and means to determine the magnitude of this control signal that is applied to the data transfer member. The elements of these control means are also physically connected together.
Each of these connections requires a significant amount of space. Consequently, as the number of neurons in a network increases, the area of an integrated circuit required simply to physically connect together all the parts of the network increases rapidly and this severely restricts the density of the neurons themselves in the circuit.